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(54) Tracking of motion of a pointing device using a single buffer for cross and auto correlation 
determination 



(57) Cross correlatton between a reference image 
frame and a comparison Image frame determine the di- 
rection of motion relative to x and y orthogonal axes for 
a pointing device (42,60) that uses optical innaging to 
fnonitor nrwvement relative to a surface (62). Pixel data 
fora portion of the surface are loaded into a buffer mem- 
ory (1 22) that shifts the data between successive posi- 
tions in the buffer memory as each pixel of a comparison 



frame is processed to compute cross correlation. Auto 
conelation is detenmined for positions in the reference 
frame and used with the cross correlation results to de- 
termine a sub-pbcel interpolation for the movement of the 
pointing device. A new reference frame is loaded using 
data for the comparison frame currentfy being proc- 
essed if the pointing device is nwved sufficiently so that 
the next comparison frame will not overtap the existing 
reference frame. 
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a memory. 
Background at the Invention 



[00021 A -mechanlcar mouse is tha most common type of pointing device used for ^^'^^'^^^^f^.f^^^^ 

f pS) SStTbus mo^ port, or a univoreal serial bus (USB) port and used lo control a cursor or 
^rS^ on rJsSavlCTMn Amouse also usually Indudos two or more switches that produce state 
° .niJSjS^dbyaJ2^and^3^nldr?L.y 

contents Sb display window and/or other JJ-f Ajf^^^ "J 
a state svmcn J emoloys a ball to actuate rotational encoders, but the ball is exposed on the 

JtTj^n^^lVBD^ source to Oluminate the adjacent surface over which the mouse is being moved. A two- 
!ie mouse, so that the signal sent to the computer for use in conUdling a cursor of ottier portwns of a d«play In 
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response to Ihe movement of the optical moose is genefally equivalent to that produced by a mecharJcal 
Z^nx sigr«I is fom-Ued to conr«cl to a PS« port or a USB port The maximum veloaty of mouse movement 
thatthlspriorartoptical8en»[Bosystemcan8ccu»ate(yti»cl«l88liout18lnches/sec. .K.-w.„«»,«n 
S)5 K,B .^nftion sysln» typicaHy use ooss comtetion and auto c«nelation to ''^termlne thejJte^en 
s of an object relative to a background. Such systems also normally usa two memory arrays, one to » «^«»roe 
SLe liBme and anotherto store a compartson image frame. Just as In the optical mouse «^f."«y ^/'V** "^J 
COToration To reduce the cost of an opUcal mouse, tt iwuld be preferable to emptoy an application speafc Inleflraled 
SoUlC) processing chip v»«»in the moose that does not require t»« distinct memonr arra^ 

{:3SSL^Z^«sedtodeten^ 
10 ance by wli^isuch an optical mouse and desirable to determine the movement of the mouse using subs>«el interpo- 

torn Aceordlngly.therelsdeartymothrationtodevetopanewappioachfbrdetemilrilngco^^ 

desired moreeffidenl tracking capabBity. with less demand for processing cydes. and at a lower cost than U« 
oirrentTpcoach. This- ne^ppmach should also achieve a tower cost by using only a single buffer memory rather 
15 lhan the two memory arrays normally required in calculating corralation. 

Summary of the Invention 
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[00071 In a pointing device that employs an optical imaging sensor for monitoring movemen relative to a surface. 
suSsslv; fnSge fnines are compaid to determine the direction and extent of the mo>«mont The ^^^J^^ 
SSuceTa sigral corresponding to pUel Image data for the adjacent surface. Inwge data for a reference frame are 
STa spSlted memSy buff^ use in detem,ining a cross correlatton wim a ^-^-^^T* 
sMuenUy ac^ by Imaging the surface. Successive pixels of the comparteon frame are processed to detemilne 
cSXnJSthe .Sss c^rSaUon as the reference frame data are shifted through the 

^t of the ooss correlallen represents the absolute value of a difference between a pixel «« ^XHlS 
pixels around a related pbcel in the reference frame, and these components correspond to the various different 
Z^il which the relaU« movement might occur. The aoss correlation component hav.ng the m«..mum value 
indicates the direction of the relat^e movement . ^ ^ 

[0008] |„aoonven«onalappioachtodetem«ninfllherelativemover,«ntbelweenanobj«^^ 
a sepirate memory amv would be provided for storing successive comparison frames and for '^^S » 
tramTHowever. in the present invention, only the reference frame is stored In memory. '^J'^^,^^ 
pointing device is substantially reduced since it is not necessary to provide a memory array or the compar«on f «me_ 
Kead.rSage data for tli successive comparison frames are processed a pixel at a time, as the <lata read 
from the Imaging sensor, so there is noneed toretain the image data for a comparison frame - unless usedtoprov.de 

" fomrSiTeSnThvention also includes the step of determining an auto correlation "«'"9 "l^^e *Jta ^^^^ 
eference frarne stored in the memory buffer. A sob-pixel interpolation of the relative movemen is then detemiined as 
aSoftheauto(andaoss)co,retoUon.Also.slnceautocorrel8tionisdetemynedinp^^^^ 
orocessing of the image data Is veiy efficient. As each pixel in the Image data for the companson frame read froin 
S^^S VenS buffer me,^ is selectively rd^^^ 

«fsj^ therein. The image data for the refeienca frame are shifted through ^uter '^mory as w<««J^ 
SiJs of the comparison f«me are read from the imaging sensor. However. 

U,r memory wiuTa new reference frame using the image data for the comparison frame that te currently be^read 
from the Imaging sensor. Since the relative movement between the pointing device and the surface is ^"torod ovw 
S.mevefSoftherelafive movement Isreaday calculated in the pointingdevlce.^^^^ 
SSlJSgsdaS^from the comparison frame as necessarytoensurethatameanlngfW 

Sedm subsequent frames. The need to toad a new roference frame «to the buffer memory Is determined asa 
fT^oiormeniocityof motion. Spedncally. the relative posW^^ 

to the velocity to the current position. For example, the position one frame aw^ can be «PP'"«^«*,J'' "^^^ 
vetodty to determine the distance that wiU be fraveled during the time for the next frame to be acquired to the amrt 
^n. The intent is to load a new reference frame into the buffer memory before the relatn« movement between 
ttM> pointing device and the surface Is sufTident to cause a subsequent comparison frame to no tonger overiap the 
reference frame previously Stored In the buffer memory. r ,k- rt«ta «f « 

[00101 Additional temporary storage is provided in the buffer memory for the last port»n Image date of a 
SrevL referencefran^so that the last portfonof the image date of the previous refe^^ 
for use m determining the aoss eoaelation as the image date for the current reference frame 
memory to become the new reference frame. Image date for the previous reference frame are discarded as trw new 
reference frame is loaded. 
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[0011] Another aspect of the present Invention is directed to machine readatXe medium that stores a plurality of 
Etive^fa"S^!^ni^ntl«^eenatleastapo,tto^ 

Z^lLudas an Imagino detector that pioduces a signal corresponding to an image of the adjacent surface, an 
ei^^s^liSr?-? .ntS^ rec^^'e the s 

etecronic swiiwi i« ^ . ^ a©nerairv carrfes out tho steps of the method described above. 

SiySfSSeVa^om^^sS^^r^^^^^ 
SSneStSre^ative'lSrementW^^ 

toapSrted image of the surface Is produced, and a reference frame for the surface that indudes dateH^or each of a 
S.X3Xe3thepixelated image isstored in memonr. AS relative movement ^^^^ 
JdStlSe a conjarison frame is produced that includes data Ibr a plurality of pixels b. a new Pb<«'ated -^f; 
SSs Z^;SrSween the data for the reference fhune and the data for the comparison ftame. es well as an 
I^^lS^?5reKlntherofe,enceframe8mthendetennlned.F^^^^^^ 
ofme^Bve movement is determined as a funeOon of t»th the cross correlation and the auto correlation. 

Brief Description of the Drawing Figures 

Th. (»»«Atnn atiMcts and many of the attendant advantages of this invention wOl become more readily 
IpplltSalrsLrSSe^t^^ 
coi^uncUon with the accompanying drawings, wherein; 

FIGURE lis a WO* diagram of an exemplaiy system for a general purpose computing device in the fom, of a 
conventional personal computer and a pointing device that implements the present inventon: ... 
FIGUI^E 2 apXily cut^y side elevattonal view of an optical mouse that employs the present invention. 
FiSS 3 Is a bl^diagram that Illustrates the component used within a pointing device to determlno 

RG~t at^^'^^rcT ^ei^^^ processing c.cu«ry used In a pointing de>.ce in accord vnth one 

dS^rwarSeranceframeofasurfaceforuselncalcula^^ 
relative to successive pixels of Image data from a comparison frtme of the surface: 
FIGURE 6 is an expanded view of a portion of the buffer memory and temporanr storage of '^^'f^^f^ 
F GURE 7 is a bliSc diagram showing a corralator state machine and the signals .np^^ 

FIGURES ami 8B are a flow chartshowlngm^ 
relative movement between a pointing device and an adjacent surface; 

P.S^eru^'.Se'S^^n'ifr^^ 

s« an rfffltkal mouse is moved over the surface at a rate of about 2 pixels/frame; 

RGUI^ !^ilSes a^inoe frame, a c«mparison frame, the disposition of a correlation window In^ 
Senirframe. and the disposition of a conflation window in the "mpajrton frame, for each of four su««ss.ve 
r^mrtarUiAn frames as the mouse Is moved over the surface shown in FIGURE 9b; , . * 

RGu'^lS^^^yir^liareferance frame stored in the memory buffer and W^^^^^^ 

a Siefatlon wJ^ow. as well as indicating the limits for other possWe locations of the correlat»n window m the 

FiaSSriSTb a schemauc diagram of a 3x3 anay of pixels. Identifying the pixel posifcns used In detenrtnlng 
the components of auto cofxelation for each successive pixel In the reference frame; and .^^^.^ 
RGuS^BIsaschematicdlagramidentifylngthepUelslndudBdineachofthefourquad^^^ 

the equation that Is employed for calculating sub-pixel Interpolation. 
Description of the Preferred Embodiment 

Exemolarv Operating Environment 

room FIGURE 1 and the following discussion provide a brief, general description of an exemplary compuUng envl- 
ronSnt Sch J^enl^ntlon may be employed. Although not necessarily limited ^'^^^J"^ 
fte^enUnvention win be described In connection wHh Its use as an Input device for providing a user-controlled 
Spu^^SnSlSaSriSLlSnal can be acted uponasdefi^ 
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modules, that ate executed by a PC. Generally, prooram modules Indude appllcalioo proflrams. routines, ol^ecls. 
components, functions, data structures, etc, that perttonn particular tasks or Implement particular al)stracl data typ^ 
Also those skilled In the art win appreciate that this inventk>n may be employed In a user Input device applicable to 
other processing environments, such as a dient device for displaying a Web page, hand-held devices, pocket personal 
s computing devices, digital cefl phones adapted to ex cute application programs and to wirelessly connect to a netvrark. 
other mlaoprocessor-based or programmable consumer electronic devices, multiprocessor systems, network PCs, 
minicomputers, mak^frame computers, and the like. w. t i ^ « 

[00161 With reference to FIGURE 1. an exemplaiy system with which the present invention is usaWe Includes a 
general purpose computing devfce in the form of a conventional PC 20, provWed with a processing unit 21 , a system 
io memory 22 and a system bus 23. The system bus couples various system components hduding the system memory 
to processing unit 21 and may be any of several types of bus stiuctures, Including a memory bus or memory controller, 
a peripheral bus. and a local bus using any of a variety of bus architeclures. The system memory Indudes read only 
memory (ROM) 24 and random access memory (RAM) 25. A basic input^output system (BIOS) 26. conteining the b^c 
routines that helps to transfer informatten between elements within PC 20 su* as during start up. Is stored In ROM 
IS 24 PC 20 further indudes a hard disk drive 27 for reading from and writing to a hard disk (not shown), a magnetic disk 
drt^e 28 for reading from or writing to a removable magnetic disk 29. and an optical disk drive 30 for reading froni or 
writing to a removable optical disk 31. such as a CO-ROM or other optical media. Hard disk drive 27, magnetto disk 
drive 28 and optical disk drive 30 are connected to system bus 23 by a hard disk drive interface 32. a magnetfc disk 
drive intei^ce 33, and an optical disk drive interfece 34. respedrvely. The drives and their assodated compute read- 
20 able media provWe nonvdatile storage of computer readable machine Instiudtons. data structures, program modules, 
and other data for PC 20. Although the exemplary envfronment described herein employs a hard disk, removable 
magnetic disk 29, and removable optk:al disk 31. It will be appredaled by those sWlled In the.art that other types of 
computer readable media wrt^lch can store data that are accessible by a computer, such as magneUc cassettes, flash 
memory cards, digital vkJeo disks (DVDs). Bomoulll cartridges. RAMs. ROMs, and the like, may also be used In the 
25 exemplary operating environment. ^ , j. ,. orv^* * 

[00171 A number of program modules may be stored on the hard disk, magnetic disk 29. optical disk 31 , ROM 24 or 
RAM 25. Induding an operaUng system 35. one or more appltoation programs 36. other program modules 37. ar>d 
program data 38. A user may enter commands and infbnnatton Into PC 20 through input devices such as a keyt)oard 
40 and a pointing devfce 4i Pointing devfee 42 is one embodiment of the present InvenUon and Is prrf«fahly a mo* i.?e. 
M although the present Invention can be implemented in other types of user Input devices in whid> at least a portion of 
the device is moved relative to a surface, such as a trade ball, a joystidc or a stylus. Other input devices (not shown) 
for PC 20 may indude a microphone, a game pad. a sateilite dish, a scanner, or the Uke. These and other Input/output 
(UO) devices are often connected to processing unit 21 through an I/O device Interface 48 that is coupled to system 
bus 23 The lerni I/O device interface is Intended to encompass each interface spedfically used for a serial port a 
35 parallel' port a game poet, a keyboard port, a PS/2 port and/or a USB port. It Is also contemplated that the present 
invention may be coupled to the PC system bus via a wireless Rnfc. A monftor 47 or other type of display device is also 
o6nnoctedto6y8tembus23vlaanappropriataintBrface.8uchasavideoadapler48,andi8usabletodispt^^ 
programs Web pa£^s. and/or other Infom^tlon. The presem invention is typically used for co^^ 
and/or controlling other graphic objects displayed on monitor 47. In connection with operaUng system and/or apptlcatton 
programs that are executed on the PC. In addition to the monitor. PCs are often coupled to other peripheral output 
devices (not shown), such as speakers (through a sound card or other audto interface - (not shown)) and printers. 
I0018J As indicated above, the inventton may be employed for use In provkJing Input to a smgle machine. However, 
PC 20 optionally operates In a networiced environment, using togfeal connections to one or more remote computers, 
such as a remote computer 49. to which the present invention can also be coupled for user input contrd. Remote 
computer 49 may be another PC. a server (whidi Is typically generally configured much like PC 20). a router, a networic 
PC a peer device or a satellite or other common networtc node, and typically Indudes many or aU of the elements 
described above In connection with PC 20. although only an external memory storage device 50 has been illustrated 
in FIGURE 1. The logical connedions depicted in FIGURE 1 indude a local area network (IAN) 51 and a wide area 
networtc (WAN) 52. Such networking environments are common In offices, enterprise wide computer networks, intran- 
50 ets. and for coupling to the Internet . ^, . * 

10019] When used in a LAN networicing environment PC 20 is conneded to IAN 51 through a networtc interface or 
adapter 53. When used in a WAN networtdng environment PC 20 typically indudes a modem 54, or other means for 
establishing communications over WAN 52. which may indude the Internet. Modem 54. whid> may be internal or 
external, is conneded to the system bus 23 or coupled to the bus via I/O device interface 46, i.e., through a serial port 
55 In a networiced environment program modules deplded relative to PC 20. or portwns thereof, may be stored in the 
remote memory storage devfee. It wtti be approdated that the network connedtons shown are exemplary and other 
means of establishing a communications link between the computers may be used, such as wireless oommunicalion 
and wide band network links. The present invention is thus suitable for use in provWing user conlrotted Input to a PC 
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or other computing devices In either a stand alone or a network confifiM«tion. 



Exemplafv Pointing Device 

[0020] Whae the pfesent hvention is not Bmitad to a pointing device, since it can be used for <»«t^'J "''J^f 
mo«ment between a user-controded object and an adjacent surface in other devices, in its initial prefened en«bodw 
mSJZ pnst^tl^n^ is used in a Lose 60. As sho«n in FIGURE 2. n»use 60 is of the optical t^^^l 
ILopUca^ imagina system for detemuning nwwwnant of the mouseover an undertying surface. Mouse ^ Includes a 

thS is Stwraay convex-Shaped to fit within the paim of a UM^^ 
wSrJstape and appearance as shov« in RGURE 2 has been soid by Microsoft Corporatwn In tho past, although 
r^^,rbuslySddidnolindudethepn«entlnve^^^ 

underlying surface K. which Is not required to have any specific design or features. Perhajw the only fe' 

rSrfa«tethat it nit be 60 honwgenoos that any portion of it appears virtually Iden^^ 

ttb^SSbie that surface 62 not Indude lines that extend ganeraily parafleK su* as the gra^n in certaU, wo^ (or 

artifidal wood) surfaces, since for any component of the motion of the mouse directed along the grain lines, it may be 

difficult to detect differences in the Image of the surface. . » -K~„«i«ciriiQP9 h«.«ina 

100211 Housing 64 Includes a base 66 that Is red. but generally optically transparent As shown in FIGURE ^housing 

SirSUaS.^elec»button68andariflhl.selec*bu«on(notshown^ 

Anadditionalbutton70 is provided on the leflsida and islypicaBy programmed to cause an Inter^^^ 

to S up te.. to dlsplayVprevious Web page. A simaar button (not shown) Is Included on the omer side of hou .ng 

M faj aS^aiSi^to and displaying the next Web page (of those previously displayed in the b«>wdr»g progran^^ In 

aS*toS^urLt.biawh^lV2l mounted in^ 

ST a; application proj^m or depressed to actuate another scrolling function. A ^^^^^'^^J'J^^^ 
and indudesapluraiity of leads(not separately shown)that communicate signalstoanVO port of PC 20 or of a^^ 

"XKn ta'p^'uRE 2. housing 64 Is cut away to schemaUcaBy illustrate a mount 76 In which are disposed 
a liQht-emitling diode (LED) 78 that emits red light directed downwardly to surface 62. The angle oftodtence ol the 
Sht — bVLED 78 onto surface 62 is relati^^ 

Z a Sxelated ImaSe sensor 80 disposed within mount 76. The light reflected from the sur^ae* ,«sses m^^h a le^ 
82 ttSfocuses the light on the puelated image sensor, in this preferred emb«Hj^ 
fte complementary metal oxlde6emloonductor(CMOS)type.«)nrigu 

«,upte<» devlce^O) array pixelafed Image sensor could be used. Pteelated image senscjr 80 produces anouU»^ 
SrSf owresponding o an ImagTlrame of a portion of surface 62. and the image data '^^'^J^^'^^^^^"''^ 
^rtlTI^gesensorfor processing in accord with the present invention, as ^^^^ 
iTproSng occurs is so great that it is possible for the presort 

flr«5^n 37^es/sec. in contrast with a previously sold optical mouse that could only track movement of the 

s^rs'JTofrrris^rnrj^^rdto.^^^^ 

of the mouse O-e.. the X axis is nominally generaBy aligned with the width of the mouse and the Y axis is nontinaHy 
genrranyaligrfed.;ilhthelengthofthemouse).buttheorientationofthese^^ 

embodirnent virtuaBy aO processing required to produce an output signal indicating any Ax and Ay components of 
^:!Sr.rfrZ^<Lthe«:rfa^.relativetotheseort^^ 

The signal conveyed through cord 74 Indicates the movement of the mouse over surface 62 wUh a resolutwn of ap- 
proxJately 400 puols per inch. Also, this preferred embodiment captures Image frames of the '"'^'^''l^^^^ 
Zat^tSOOoLmesLc. An alternative embodiment. Which IS not Shown, em^^ 

to PC 20 or another type of computing device by a wireless link. It is expected that the wireless mouse wIU capture 
ImageframeswiththepixelatedlmagodeteclorataiateofabouteOOOftames/sec ™,„,„„ .««,.™,„t 

f00241 Referring to FIGURE 3. an overall block diagram Ulustrates the functional btocKs for detem.Ining nwvement 
to produ« an output signal from mouse 60 in response to a user-controlled movement of the mouse over surface 
The analog signal produced by the pbtelated image sensor Is read from the detedor row by row. pixel by pixel in each 
^.Tte?9,^lsSputtoanaWt<Kligital{A4))converterM^ 

Sl^^giS with a roscutton of eight bits/pbcel. The value of each pbcal in the 20x20 image ""^^.^ "^^^ 

between 0 and 255. corresponding to a gray scale pixel for the image frame. The light sour« and 

are controlled so that an average value for the pixels in the 20x20 image frame is equal to about 128 (i.e.. one-half of 

iMSl^Tt^ 2l signal produced by A-O converter 90 Is then Input to a navigation block of a processing circuit aa 
E in a tunrtional Lck 92. The processing circuit detemiines a cross correlation between each successive p«el 
within a correlation window of a comparison frame and a range of pixel positions around a corresponding p«el in the 
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correteUon Window Of a referenca fhime tl«it b .to«d in a buffer rw^ 

pixai the processing drcutt also detenrtnes an autoconelatkmfor successlvo pixels In the reference frame. Mswg the 
Sm^n^^SS c««laUon «d auto c«r«.ation that are thus determined, the processlnfl cljcuU then preduoM 
an output signal indicating movement of mouse 60 relative to surface 62. caresponding to a displacement AX antfor 
Ay along the respective orthogonal axes. As Indicated in a block 94. this output signalfbr th polnUng device indKates 
the relattre movement and Is Input to a computer, such as PC 20 or to another oompuUng device. The s^nal can be 
input through any port for which R is appropriately Ibmiatted. such as the PSa port or the USB p^ 
rSn^taUsofaLigation processing d»«i»100 are illustrated in FIGU^ 

fnm ArO converter 90 (shown in OGURE 3). An image detector interface 102 handles tinning considerations when 
reeding the digital data for each pixel Included In the digital signal as the Imago data are read from the image detector 
as successive pixels, row by column. The pixel data from the image frame are then processed through a highiwss 
filter 104. and the resulUng filtered data are Input to a correlator 1 06. Correlator 106 includes a buffer mer^V in -J*!^ 
the pbel data for a reference Image are loaded and temporarily stored, as explained further below, and a MUX that 
determines whether th& previously loaded reference frame Is reloaded ba<* Into thebuffermemory. or date to 
a cuirent comparison frame are loaded into the buffer memory as a new reference freme. The state collator 1 06 
is contiolled by a correlator state machine 108 under the control of a pointing device posiUon calculator 110. 
[0027] The output of correlator 106 Is Input to pointing device position calculator 110 tor use In ^Wfnn'n'^S the 
direction and extent of the relative movement between the pointing device and the adjaeent surface. wrtWn the hinrted 
renoe of the cross correlation. Also Included within pointing device position calculator 110 is a fractional pbtel calculator 
11 2 that uses suto correlation and cross correlation to determine a sub^)ixel Interpolation as part of the •nwement 
sLnToutput from the navigation processing cira.it A movement signal output Intorfaoe 114 that « coupled to the 
pointing device position calculalor appropriately fom»ts the movement signal for Input to a selected type of Input port 
on a PC. This movement signal includes any Ax and Ay integer and subijixel components. The ^^ptol component 
for each axis is detemiined by fractional pixel calculator 11Z When fomtwtted for Input through a PS«. USB. or other 
appropriate port of a computing device, the output signal from mouse 60 can then be used for controlling the posAon 
of6«jrsororolherobjectonadisplay.lnresponsetoinputfrom8useroontrollinglheroouse. 
[00261 To avoid edge effects. Image detector Interface 1 02 strtps away the first and last row and the first and last 
cduinn of the 20x20 pixel image frame, providing only digital Image data for the Innemost 18x18 pixeb of the ongir^ 
20x20 pixel array to hiflhoass filter 1 04. Accordingly, only pixel dato from an image frame coaespontfng to an iftxiR 
pixel array (centered wittiin the 20x20 pixel Image Irame) are toaded into the buffer memory of ooirelator 106 for each 

M02M**liI'R6UPE5!"details of a buffer memory 122 are illustrated. Buffer memory 122 receives successive pixel 
data from a MUX 120 having a first Input 120a and a second input 120b that selectively are enabled to provide tt» 
pixel Image data to buffer memory 1 20 tttrough an output 1 20c of the MUX. The reference frame previoushr stored in 
U« buffer memory are retoaded Into the buffer memory in responseto8LOA0_REF8^nalprovldedbyM^^ 
machine 108 (FIGURE 4). Alternatively, the pixel data for me cunent comparison frame are loaded ""to «^f '>y™[ 
memory for use as a new reference frame. MUX 120 thus.sen^es as an electronic switch to detennne whether the 
pixel data previously stored within buffer memory 122 are reloaded, which Is the normal case, or whether a new ref- 
erence frame Is loaded using the filtered pixel data signal for the current comparison frame. 
[00301 A key benefit of the present inventton relates to its abflily to detemMne movement based on cross correlaHon 
that is computed using the pixeidatafor the referenceframe stored in the buffernwmory. and pixeldatato 
trame. which need not be stored in a memoor anay. Data for successive pixels of the comparison frame are input to 
the pointing device position calculator for use in connection with appropriate corresponding data for the pUels ttored 
within buffer memory 122 for the reference frame. The stored data for the reference frame and the incoming data for 
the comparison frame are then used to calculate components of the cross correlation fa each of a plurality of different 

components related to possible directions of movement . a 

[00311 In this preferred embodiment, cross correlation is determhed only for a speafic 12x12 pixel porBon (te. a 
correlation window) of the total 1 8x18 pteel array for both the reference frame and the comparison frame. The spe«3fic 
12x12 pfacel portions of the 18x18 pixel reference and comparison frames thst are used In the cross correlation deter- 
mlnatiwcorrespondtothepUels within th8l2x12eonela«on window location in each Ima^^ 
the manner In which the tecatlon of the 12x12 oorrelation window is detennined within the reference frame and within 
the comparison frame are explained t>elow. • ■ . 

[0032] As the date for each pUel of the 12x12 correlation window in the comparison frame are read one pixe at a 
time f^om the pixeiated image sensor, the components of the cross conelation array that uw ^^at date are «ta.lated 
by detennining the absolute value of U»e differences between the date for lhat companson frame pixel and the relevan 
date in the oorrelation window of the reference frame. It Is the processing of the comparison frame pixel date one pixrt 
at a time in the present invention that enables cross correlation to be determined without the need for storing the 
comparison frame In a separate memory buffer. 
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10033] in addiUon to determining cross eorrelation. the correlator also determines auto <=^^^J^^J^ 
Ethe correlation window of the reference frame. It should be noted that unless a new reference f^me te load^ 
ri^o,y122.lhe auto chelation .omponentsof the reference 

embodiment auto correlation is recalculated each time that cross correlatwn is detennlned. roflanfless «' JM^ethera 
SrireS^^te..SWt»en loaded into buffer memory 

ZTo^SS pixel cw^^ 

SteErS?ela2n. Ho\vever.sim»theautocorrebtk« 
SSSs^al to zero, there is no need to calculate n. and thus, only eig^^ 

SZrR«ther details relating to detemilnation of auto correlation and to disposlUon of components of the auto eoire- 

SSrin^^S^S'uding 18 row, 124 of data for each of 18 different pUels. buffer memory 122 a'^olndudes 
^^ZT^toS^26 throuflh Which data for a previous reference frame are 8W«ed a-jd stor^for u« 
ffH!S^i™^c«n9^aH«^ as a new reference fr^^ It should be apparert that 

^rSS^StpUels'anewreference 

t^lolte c^Smlallon using data fbrthe previous reference frame, temporary storage 126 is required to^ore the 
STthTSeTsTu«?a^^ rows of the previous reference frame. As the data for the new "'erenca frame «e 
it^lTil bSr memory 122. the data for each pixel position are shifted to the left until upon teaching the far left 
^^^^ll^PE5Wle^\n regards to the llustrated example), the data are then shifted to the far nght e|>d 
fJriS^iSe^^nm^Sl^^^ 

l*SdtlS,ughtemporarysto.age126apixelatatlm8.sothatasthedatafbrthenew^^^^ 

me uDoer left^ of fte 1 8x18 array, the data for the last 38 pixels of the old reference frame are being Rifled Into 

teZS^ sSiS^S ALxdinflly.tL data for the prevk^^ 

4KniSto^«aSS 

eSer memo.v 1 22 has certain unique characteristics that are m some ways similar to a shift regfeter since 
Se data S eirSSxel is shifted through the buffer memory generally in the same mariner " tt«;o"8h a 
^r^Zr mZ^P^ for temporary storage 126 and the ability to read data from selected portk«« of 
S^S^^nSTforise in calculating cross and auto correlation are unique to the design of buffer '^'^^^ 
raoaq FSS^eSluslratesdetaiis^theuppertwon^ofbuffermemoryl^ 
inS^refUnceframe has been completely loaded into bulfermemo»y122.thex.ycoordi^^^ 
err.X llTTn ttTlBxTs pixel arri are generally as indicated in this FIGURE. When the reference ftame Is fully 
^^tl ^bSer «mo1y. a pix J havi^ x and y positions 0. 0 will be "P'-'fX:; 
memory 12Z adiacent to a temporary cell labeled TO In temporary storage 126. As the first pixe '^^^^^f 
^JJson tame are then loaded into the correlator, the image data for pi^^^ 
2S^Xinput120b.foruselnreloadingbuffermemo.y122withthedataforcurrent«^^^ 

X Dm^a.sSeret«; frame data wHI not happenifsuffi^^^ 

SE^d^S rSer^r^eframe into buffer memory 122. A new reference frame must be loaded to avoid a srtuat«n in 
Stared nSntTeriap between the comparison frame next to be 1^^^ 
5oSl TsS,ul5rie^dentthMas 

iusid to predict When the next comparison frame to be read will no longer overiap wrth a ^f"^^^"!"" ^ 
» th7t ft will be necessary tokMd a new oompaiison frame Into buffer memory 122 as the new «^^"«* fr^T 
riineiSnTtStod^new reference frame. MUX 120 selects first input 120a instead of second inpu 120btolcad 
^ S^SSSTpaSon frame into the buffer memory. In any case, data for each pixel of the current "ferenceftame 
«eSredir;eJI^rystora9e126andafterbelngshifled^^^^ 

posT n'^S ISJtes the various signals that a» input to and output from correlator ^'-te r«d,h« m 
iWtoconelatorstatemachinelOSareprovidedbypointingdevice position 

,0 the input signals. FILTER.VALID is an Input signal IndlcaUrjg whether the output f^^Pf« ""^^^ 

correlatw are valid data for a pUel position of the cunent Image frame. The Input signals REF_X_START and 
^?^T/StT^pSely. refer to t^xandystart addresses w^ 

SwSinSrra= X E^^ Y ENDrespectivelyrefertothe end addresses fbrthecorretatorwindowin 

Zl^-reSJSllt lour inpu. ^^^s are the x and y start and x and y end address J« ^-P-'-; 
frame currently being processed. Another input signal. NEW.FRAME. indicates to the correlator state madiine that 
rrriS^S;paS,S,Smehasbeen imaged and isavailableforinput.apixel at 

correlalton with the reference frame Stored In the buffer memory. ^ ... , .^H«^K»an 

S Cont,t-ofthebuffermemo,y.causingittoshiftthedataforeachpixeltothe™^ 
ADVANCE REF FRAHAE signal output by conelalor state machine 108. Similarty. a signal ADVANCE COMP PIXEL 
to the irUe detector Intwface to enable input of the data for the next pixel in the companion frame. The 
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calculation of cross conalation and auto correlation is initiated by CROSS.CORREIATE and AtiTO.CORRElATE 
signals. respecUvely. which are output by the correlator slate machine. .. ^ , . , , „.„^«. 

[0040] FIGURES 8A and SB Ulustrate the stops Implemented In the present Invention for detemtUning relative move- 
ment between the pointing devic and iU adjacent surface tased upon the calculation of cross correlation and auto 
correlation. In a step 140. for example, upon Ntial boot-up of the pointing device. aU required registers are InlUalteed. 
Next, the logic provides for reading a new Image frame In a block 142. using data conveyed by the output signal from 
the pixelaled Image sensor. Since a reference Itame win not Initially be available m the buffer memory, a block 144 
provides for loading the image data Into the buffer memoiy. After this Initial reference frame Is leaded, flw next image 
l!Bmereadbacomparisonftameimage.endltsdataareprocessedpb(elbypUel.aspro>ri^^ 
blocic 148 determines if both the current reference and comparison frame pbrels are valid. This determination is afflrm- 
aUve if the current pfeel is within the cofretatlon window, which is a specific 12x12 pbcel portion of the 18x18 pixel 
comparison frame and 18x18 pixel reference frame. Initially, the correlation window is centered In each of the reference 
and comparison frames. If not a valid pbcel, the next comparison frame pbcel is read and pixels for the reference frame 
are cyded by shifting Item to thefr next position 80 that tl» corresponding next reference frame pixel is used In a step 

150. The test at decision step 148 is «ien repeated. 

100411 InHially. as noted above, the correlation window is centered vwthin both the comparison frame and the refer- 
ence frame. I.e.. with a threo<»Iumn and three-row margin around each side of the correlation window in the 1 8x18 
pixel array portions of each of these two frames. (Note that a margin of at least two rows and two columns sunounds 
Ihe correlation window In the reference frame and comparison frame.) , . w • 

100421 In a step 152. the correlator calculates aU of the cross correlaUon coefficients for the eun-ent pixel being 
orocessed. As noted above, there are normally nine cross correlation coefndents for a phel. However, under certain 
drcumstancas. the size of the correlation array Is increased from 3x3to 5x5. to provide foratotal of 25 cross COT^ 
coeffidents. The reasons for increasing the size of the cross correlation array are discussed below; however. It is only 
under unusual drcumstancas that ft becomes necessary to Increase the size of the cross correlation array In this 
manner Following the calculation of the nine cross correlation coefficients (or 25 coeffidents when necessary), a step 
154 provides for calculating all of the eight auto correlation coeffidents for the current pixel in the reference frarne. 
Next, a decision step 156 detennines if all of the valid plitels In the correlation window have been processed, and 
not, the logic loops bade to step 150. ^ . 4co ^ t 

100431 After all of the valid pixels for the current comparison frame have been proca-wed, step ^fs^ p'O'OOes tor 
determining the correlation wireJow positions for both the reference frame and the next comparison frame to be proc- 
essed. Detafls Of the logic applied In implementing step 158 are discussed below. 

[00441 At a step 160. the correlator begins reading the next comparison frame, a pbcel at a time. Based upon ttie 
correlation window posMons in the reference frame and comparison frame that is now being read, a step 162 provides 
for detennininfl the x and y start positions within the 324 pixels positbns (or each of the reference and coinparison 
frames. SimUariy. in a step 164. the x and y end poslttons ft)r the 12x12 correlation window are detemUned. Based 
upon the relative start positions, the offset is determined between the assodated pixels of the reference frame and 
oempariMn frame in a step 166. The start and end positions within the reference frame can differ {be offset) from those 
In the comparison frame byes much as two rows and two columns, for the respective 18x18 image anays. 
(00451 A decision step 168 detennines If the olfeet that was detennined is positive, and if so, the logic proceeds to 
a decbionstep 170 thatdetemiines the eomparisonand corresponding reference pixels are aligned. This stop actuaHy 
determines ff the appropriate comparison and reference frame pixels ate being used (or purposes of detennlning cross 
eoaelation. If not. a step 172 provWoafor shitUng to the next reference frame pixel before repeating the determination 
in dedsion step 170. Once the reference and comparison frame pixels are aligned, the logic proceeds to a step 178 
in FIGURE BB 

[0046] If the result In dedsion step 168 indicates that the olfeet is negative, the logic proceeds to a dedsion step 

174 In FIGURE 88. This step again detennines if the comparison and corresponding reference pbcels are aligned, and 

if not a step 176 provides for reading the next comparison frame pixel. In this manner, the logic ensures that ojrre- 

sponding reference and comparisonframepteelswilhinthecorrdationwindowareusedftjrcomputing^^^ 

Followir«thedetenninationineithefdeddon8tep170or174.step178calculatesalloflhecro^ 

or components fc>r the current pbtel. SImilariy, a step 180 provides fcir calculating all of the auto correlation eoefltaents 

or components for the cunent pixel in the reference frame. Adedsion step 182 detemnines if all of the valWpbcels tove 

been processed and If not. a step 184 reads the next comparison frame pbral and cydes the reference fr»no so that 

aIloftheposrtionsforthereferenceframedataare6hHtedbyonoposition.Step178thenrepeats. 

(00471 After processing all of the valid pbcels. a step 188 MenUfies the location of the maximum wirelatlon pUel 

among the nine components for cross conelation (or on those occasions when necessary, among the 25 componetrts). 

The position of the maximum correlation Indicates the direction in wrtii* movement has occurred between the relative 

position of the reference frame and that of the comparison frame. Accordingly, K is possible for movement to have 

occuired in any of the eight possible diredions, or alternatively, for no movement to have occurred, in virhidi case the 
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maximum correlation wOl be <n the center posHion of Ihe correlatioo array. 

[0046] A step 188 then determined a 8ut>-pixel interpolation based on auto correlation (and cross correlation) com- 
ponent. The combination of the dirocUon in which maximum correlation occurred and the $ut>-pixel interpolation deter- 
mines the component for ^ and Ay in block 1 90 that wUI be output from the navigation processing circuit for use by 

« the computing device to which the pointing device i$ connected. A Wock 1 92 also provides for determining the velocity 
of the pointing device over a surface as a function of the accumulated nnotion in a given direction over a defined time. 
Based upon the velocity, a determination is made as to whether it is necessary to load a new reference frame into the 
buffer memory using the next comparison image that is processed. This determination ensures that sufficient overlap 
will be provided between a subsequent comparison frame and the reference frame in order to determine the direction 

fo of movement of the pdnttng device relative to the adjacent surface. If it is necessary to load the next comparison frame 
into the buffer memory for use as a new reference frame, a step 196 provides the signal to MUX 120 to cause it to do 
80. Otherwise, from decision step 194, a step 198 begins reloading the refererKe image data Into the buffer momofy 
for use in again determining the movement of the pointing device relative to the adjacent surface. The logic then returns 
to step 1 58 in FIGUI^ 6A, to determine the conelation window positions in both the reference frame and the coaelation 

i5 frame.' 

[0049] Step 1 58, which pnwides for deterniining the correlation window position. \s easier to understand In regard 
to a simple graphic example. In FIGURES 9A and 9B, a graphic image 210 represents the surface-undertying mouse 
60. A 20x20 image of a portion of grapNc Image 210 occupies a region 212 and corresponds to the image detected 
by the image sensor in the mouse. Region 212 moves to the left over graphic image 210 as the mouse Is nroved in 

20 that directton. as indicated by an arrow 214 In FIGURE 98. 

[OOSO] The logic applied In determining the position of the correlation window In the reference and comparison frames 
is illustrated In regard to the portion of graphic image 210 imaged by the image sensor In mouse 60 as the mouse Is 
moved over the graphic image. A first row of FIGURE 10 fflustrates a reference frame 216, comprising 18x18 pixels 
of region 212. Reference frame 216 includes a left side of a face of a stick figure contprising graphic image 210. Initially, 

25 a comparison frame 218 in arfunvi A of FIGURE 10 Includes the same pixel data as r^erence frame 216. However, 
as the mouse is moved toward the left, an 18x18 pwel ccrnipartson frame 220 shown in column B is shifted slightly to 
the left, so that It includes slightly more of the Image of the underlying surface to the left of comparison frame 21 8, and 
slightly less of the underlying Image on the right of comparison frame 21 8. Further changes in the portion of graphic 
image 210 Induded within comparison frames 222 and 224. in columns C and D. occurs as the mouse Is moved further 

30 to the left over the graphic image. 

[W51 J Initially, in column A, a correlation window 226a « disposed within a central portion of reference frame 216, 
and comprises the innwmost 12x12 pixel region centered within the 18x18 pixel reference frame. Similarty, In column 
A, a correlation window 230a within comparison frame 218 is also centered as a 12x12 pixel array within 1 8x18 pbcel 
comparison frame 218. However, in column B. as a result of the movement of mouse 60 toward the left, a conelation 

35 window 230b is now shifted toward the right margin of comparison frame 220 so that the portion of the image in the 
12x1 2 pixel comparison conelation window generally matches that in the 1 2x1 2 reference correlation window In column 
B. 

(00521 Under column C, as further motion of the mouse to the left occurs, a reference conelation wfrxJow 226b is 
shown shifted sUghUy to the left, relative to reference correlation window 226a. The shift of the reference correlation 
40 window in this manner thus causes a relatively dose match between the portion of the graphic Image in reference 
correlation window 226b and that in comparison conelation window 230b. In the final column of this example, a refer- 
ence corretatton window 226c Is shifted to the far left margin of reference frame 216. maintaining a relatively dose 
match between portion of the graphic image in reference conelation window 226c and that In comparison correlation 
window 230b. 

45 [0053] From the precedfr)g graphic example. It will be apparent that based upon a previous determination of the 
cross correlation and the related movement the relative position of the correlation window is shifted in at least one of 
the reference and comparison frames to maximize the portion of the Image that appears within the correlation window 
of both the reference and comparison frames, for detemiination of subsequent movement as the next comparison 
frame Is processed, pixel by pUel. The change In Ihe position of the correlation window in each frame is detennlned 

so based upon the movement of the pointing device lust detennlned using the last cross conelation. 

[0054] It is somewhat easier to understand the relationship between the 18x18 pixel reference frame (or comparison 
frame) and the position of the conelation window by refening to FIGURE 1 1 . As shown in this Rgure, an 18x18 reference 
frame 240 indudes long dashes 242 that define the limit of the possible locaUon of a 12x12 correlation window 244 
within the reference frame. In the example shown in FIGURE 11 . correlation window 244 has been shiaed to the upper 

M margin and left margin of the limit line. A similar relationship exists within the 1 8x1 8 pixel comparison frame In regard 
to the possible positions of the 12x12 conelation window. Accordingly^ to the extent possible, changes in ihe position 
of the 12x12 conelation window in one or both of the reference frame and comparison frame are made within these 
limits to maximize the extent of overtap between the portion of the reference frame to which each subsequent conv 
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partson frame te to be compared. The disposition of trie correlation window within the reference frame and comparison 
frame is determined by the position of the pixel of maximum cocrelation to the reference frame, for the previous com- 
parison frame. When the velocity of the pointing device over the surface Indicates that a new reference frame should 
be loaded, the previous determination of movement stiU can be used to determine the relative location of the 12x12 
correlation window In the new reference frame and In the subsequent comparison frames. 
[0055J RGURE 11 also illustrates a 3x3 array 246 of pixel positions relative to a pbcel position 248 indicated by "P" 
at the upper left comer of 12x12 correlation window 244. The 3x3 pixel array around position 248 indicates the nine 
components for cross correlation and thus Indicate the possible directions of movement of the polnUng device relative 
to the underiying surface. As subsequent pixel data are shitted Into position 248. further detefmiiwUons of the compo- 
nents of the nine coeffcients for cross correlation aro added to the previous detenninations for the 144 pixels in the 
correlation window. Simaaity. in the reference frame, auto correlation is determined fc)r each of the 144 pixels compnsing 
the 12x12 correlation window. The cross correlation is used to determine a direction of maximum correlation (which 
corresponds to the position or direction of minimum difference between related pixels in the comparison frame and 
reference frame). The Maximum conelation position In the cross correlation array thus indicates the direction of move- 
ment A maximum correlation at the center position indicates that substantially no movement has occunwi. 
r0056J In the event that the qualily of the surface or other condHions causes the maximum difference for the nine 
components of the cross correlation between the reference frame and the comparison frame to fall below a predefined 
minimum, the logic wiU detemiine that the current cross correlation cannot be reiied upon to determine movement of 
the pointing device relative to the surface, Accordingfy. for the next comparison frame, 25 coefTidents will be determined 
for each of the 144 pUels in the comparison frame. These 25 positions are enclosed by a dotted Une 250 in FIGURE 
11 The 5x5 cross correlation array can detennine relative movement by as much as two pixels in either the positive 
or negative x d wction and/or two pixels in either the positive or negative y directioa Thereatter, the cross conation 
Is made again with a 3x3 anay of coefficients to detennine a direction of movement wilhin a -1 to +1 value for the x 
direction and a -1 to -t-l value for ttie y direction. 

r0O571 FIGURE 12A iUustrates the nomenclature used in connection with detennlning auto correlation reiath^ to a 
pixel Pn 0 at a position 262 in the center of a 3x3 pixel array 260. In regard to the equation provided below, the value 
of the variable a can range from -1 to 1 , and the value of the variable 6 can range from -1 to 1 . therefore providing auto 
correlation on the basis of nine positions from P.i through P,^, as Indicated in FIGURE 12A. The auto correlation in 
this case is relative to position 262 for each of Uie eight positions 264 surrounding position 2fi?. Sinw* the 91 rto corre- 
lation for a pixel position relative to itself is always 1, it is not necessary to calculate the auto conelation component 

for position 262. ^ ^ ^ ^ _ 

[0058] FIGURE 12B graphically Blustrates the disposition of (bur different quadrants. One of these quadrant Is se- 
lected based upon the direction m which relative movement between the pointing device and the surface last occurred, 
and the relath/e movement is detemifrted from the cross conelation. as explained above. Depending upon the position 
(surrounding a center position 280) In which the best correlation {minimum difference) was found for the auto correlation 
determination, one of four quadrants. 0. 1, 2. or 3. as represented respectively by dotted fines 272, short dash lines 
274. dash-dot lines 276, and dash lines 278. wai be identified. The quadrant in which the maximum auto conflation 
was' computed thus detemiines which of four different equations is used in calculating a sub-pixel interpolation com- 
ponent assigned Ax and Ay for the movement signal. Details of the equations for computing cross correlation and auto 
correlation are set forth befow. 

Equations for DetenninInQ Cross and Auto Correlations 

(00S91 The general equation for computing aoss conelation between a comparison frame and a reference frame in 
regard to a and b components is given by; 



cross correlaiionia. &) = J) f^|coOTpomon (x, - reference (x + a, y + 6)| 



Where, nonnally \s\ this prefened embodiment, -isasl. -Is6s1, and thus, there are a total of 9 aoss correlation com- 
ponents for each frame. Both A/ and /U are equal to 11 in the prefened embodiment 

[0060] If. due to the generaUy unlfonn nature of the surface over which the pointing device is moving, the minimum 
cross correlation component computed for the cunenl comparison frame Is less than a predefined threshold, the result 
cannot be reiied upon and will be discarded. For example, a surface area having a generally unifonn gray scale value, 
and a generally homogeneous texture might well produce a resulting cross conelaUon that fails to accurately indicate 
movement. If such a result Is obtained for the cross conelation, an alternative 5x5 cress conelation array with 25 cross 
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correlauon components is determined to find a correlation minima that is used to better align the correlation window 
with the comparison and reference frames. Using the Information from this larger cross correlation array, the position 
off the correlation window can twtter bo determined; however, for the next comparison frame, the logic again wUl revert 
to using a 3)C3 cross correlation array, with nine cross correlation components. ^ ^ , 

10061) The general equation for computing auto cornelalion components for the pfac^s of a reference frame is similar 
to the equation for determining cross correlation and is as foflows: 



autocorrelation{a,b)^j^f^^^^^^ 



Where, in the embodiment preferably used, -isa^l . -1^1 . so that there are a total of 9 auto correlation componente 
for each frame. Agaln.\)Oth N and M are equal to 11. The auto correlation component for the case where both a and 
b equal 0 is always equal to one, so only eight auto correlation components need be computed for each pixel in cor- 

mm "fw determining subi)ixel interpolatton. which provides better resolution In specifying the Ax and Ay signal 
indicathre of the relative movement of the pointing device over surface, both auto correlation and cross conrelation 
values are employed, in a two-step process. First, variables that characterize a shape of an auto correlation cun^e 
(generally concave or bowl shaped) are determined as follows: 

Sf f = 3/2 • l(AC3*AC7) - (AC1*AC9)l 



B20 = 2*({AC4MC/+AC7MC3MC6*/^C9)] 



B02 ' 2%A0UAC2^A03^AC9^ACB^AC7)] 



DENOM = IB20'B02 - B1VB11] 



where AC1 through AC9 correspond to the auto correlation coefficients determined as i>oted above. The nine auto 
coirelation components. AC1 through AC9. detemiined for the 144 pixel positions In the correlation window of a frame 
are numbered from ACf in the upperleft to ACS in the lower right, consistent with the 

of 3x3 array 260. which Is shown in FIGURE 12A. Next, a MINIMUM.QUAD.LOCATION is detemiined. based upon 
the quadrant of the auto correlation array in which the smallest auto correlation component is found, as shown in 
FIGURE 12B All four of these quadrants overtap on the center auto correlation value, position 280. which corresponds 
to auto correlation coefficient ACS. The sub pixel interpolation values for Ax and Ay are detennined based on the 
minimum cross correlation value and as a funclionof the quadrant (0. 1 . 2. or 3) In which the minimum cross corre a ton 
difference value was found, using the auto correlation parametere computed above and using selected cross correlation 
values, with one set of the following equations: 

Quadrant 0 

sub jwx_x = 4- 2 • [{B02.ef f)* (Cf-C5) + (B02*Biir (C4-Cf )] iDENOM 



sub >x y^.yi^2^HB20'B1iy{CUC5)^{B20*B1iy{C4^1)]fDBNOM 

Quadrant 1 

sub = K - 2 • l(B02-Bf f )• {03-05) * [B02^B1iy {Ce-C2)] IDENOM 
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sub j>ix _^ = .% + 2M(fi2C^^^r(C3-C5)-^(S20+Btf)MC«-C2)l/OHWOM 

Quadrant 2 

sub jiitjc^^ - 2 • [{BOZ-BUy (C9-C5) + {B02^BUy {C&OB)] iDENOM 
sub _pix^ = >5 - 2*UB20-ef f )* IC9^5) + {B20*B1iy {C6-^8)] IDBNOM 

Quadrant 3 

sub J)ixj( = -H + 2 ' l{B02^1iy {C7-C5) ♦ (Sf>2+Bt f )* (C4-Ca)l /DENOM 

sub j)fx^ = V4- 2 • KB2a^n)' (C7-C5) + {B20*B11) * (CM-C8)) /D£NOM 

where the CI through C9 components for the cross correlatfon are also \n the mdicated array pos'rtions identified by 
the numbers in the comers of the array elements shown in FIGURE 12A. 

[0063] Although the present invention has been described in connection with the preferred form of pracUcing It, those 
of ordinary skill in the art will understand that many modifications can be made thereto within the scope of the daims 
that follow. Accordingly, It is not intended that the scope of the hvention in any way be limHed by the above descnption, 
but instead be determined entirely by reference to the claims that follow. 



Claims 

1 . A pointing device that produces an output signal indicative of a relative movement between at least a portion of 
the pointing device and an adjacent surface, comprising: 

(a) an imaging detector that produces a signal corresponding to an frnage of the adjacent surface; 

(b) an electronic switch having a first Input to receive ttm signal from the imaging detector, and a second input, 
said switch being operative to selectively couple one of the first ir^ut and the second input to an output; 

(c) a sequential memory buffer havirig an input through which data are input for storage and an output through 
which data are transfwed out of the sequential memory buffer, said input of the sequential memory buffer 
being coupled to the output of the electronic switch and said output of the sequential memory buffer being 
coupled to the second Input of the electronic switch so that said input of the sequential memory buffer selec- 
tively receives data oonveyed by either the signal from the imaging detector or the output of the sequential 
memory buffer, depending upon a state of the electronic switch; and 

(d) a processing circuit that Is coupled to the Imaging detector to respond to the signal conesponding to the 
image, to the sequential memory buffer to selectively read data stored therein, and to the electronic switch, 
said processing circutt carrying out a logical process that: 

<i) selectively controls a state of the electronic switch to selectively load the sequential memory buffer with 
data derived from the signal produced by the imaging detector in a first state and with data from the output 
of the sequential memory buffer in a second state: 

(ii) uses the data in the sequential memory buffer for a reference frame and the signal from the Imaging 
detector that comprises a comparison frame to calodale a cross correlation between the reference frame 
and the comparison frame; and 

(III) produces the output signal indicative of the relative movement between at least the portion of the 
pointing device and the adjacent surface as a function of the cross correlation. 

2. The pointing device of Claim 1 . wherein the processing circuit further uses the data for the reference frame in the 
sequential memory buffer to calculate an auto correlation. 

3. The pointing device of Claim 2, wherein the processing circuit determines a sub-pixel interpolation for the relative 
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movement between at least the portion of the pointing device and the adjacent surfece as a function of the auto 
correlation. 

4. The pointing devlc of aaim 1 , wherein the sequential memory buffer shifts data for pbcds between successive 
storage locations contained in the sequential memory buffer as data from the output of the electronic switch are 
applied to the input of the sequential memory buffer. 

5. The pointing device of Claim 1 . further comprising a high-pass fitter that is applied to the signal produced by the 
imaging detector, an output of the high-pass fiter being coupled to the first Input of the electronic switch and to 
the processing circuit 

6. The pointing device of Oalm 1. wherein the electronic switch comprises a multiplexer that detemiines whether a 
new reference frame Is loaded Into the sequential memory buffer or whether the reference frame previously stored 
in the sequential nlemory buffer is reloaded therein. 

7. The pointing device of Claim 1. wherein the data stored In the sequential memory buffer are atleastpartlyselectively 
addressable and accessEile by the processing circuit on a pixet-by-pfxel basis. 

8. The pointing device of Qaim 7, wherein as the processing circuit selectively controls the electronic switch to store 
each (^xel of a comparison frame in the sequential memory buffer as a new reference frame, the data for each 
pixel of said comparison frame are also employed by the processing circuit In calculating a portion of the cross 
correlation using the data stored k\ the sequential memory bulfer for a previous reference frame. 

9. The pointing device of Claim 8. wherein the processing circuit employs data for a pUel of the reference frame in 
calculating a portion of the cross correlation and then transfers the data eiUier out of the sequential memory buffer 
so that «w data aia no longer used, or transfers the data back into the input of the sequential memory buffer 
through the electronic switch, depending, upon the state of the electronic switch. 

10. The pointing device of Claim 1. wherein tt\e reference frame stored in the sequential memory buffer is replaced 
with a new reference frame comprising a cunnent comparison frame as a function of a relative velocity between 
the pointing device and the adjacent surface, said velocity be determinative of whether a meaningful cross corre- 
lation can be determined from successive comparison frames. 

11. The pointing device of Qami 10, wherein the reference firame stored in the sequential memory buffer is replaced 
before relative movement between the pointing device and ttie adjacent surface Is sufficient to eliminate an overiap 
between tiie reference frame that was stored in tiie sequential memory buffer and a current comparison frame 
being input from tiie imaging detector, the overiap being necessary to determine the cross correlation between tiie 
reference frame stored in the sequential memory buffer and ttio comparison frame currentiy being input to the 
processing circuit. 

12. The pointing device of Oaim 1, wherein a data storage capacity of «w seqwntial memory buffer is substantially 
less ttian that required to store both tt^e reference franre and the comparison frame. 

13. A method for producing a signal indicative of a relative movement between at least a portion of a pointing device 
and an adjacent surface, comprising the steps of. 

(a) imaging the adjacent surface to produce image data; 

(b) sequentially storing tiie image data in a memory bulfer, the image data stored in the memory buffer corre- 
sponding to a reference frame; 

(c) determining a cross correlation between the reference frame stored in the ntemory buffer and successive 
pixels of a comparison frame acquired by Imaging the adjacent surface, as the relative movement occurs 
between at least the portion of the pointing device and ttie adjacent surface; and 

(d) as a function of the cross correlation, producing ttie signal indicative of tiw extent of the relative movement 
between at least the portion of the pointing device end ttie adjacent surface. 

14. The method of Claim 13, further comprising the step of detennining an auto correlation using the image data for 
the reference frame stored In \he memory buffer. 



14 



EP 1283 493 A2 

1 5. The method of Claim 14, further c<»nprising the step of determining a 8ul>-pixel bterpoIaUon of the relative move- 
ment as a function of the auto correlation. 

16. The method of Claim 14, wherein the step of determining the auto correlation is carried out in parallel with the step 
of determining the cross correlation, for each successive pixel of the comparfeon frame. 

17. The method of Oalm 13. further comprising the step of selectively reloadfrig the buffer memoiy with image data 
for the reference frame. 

18. The method of Oalm 13, further comprising the step of selectively loading the buffer memory with image date for 
a new reference frame, using the image data for the comparison frame. 

IS. The method of Qalm 1 8, further comprising the step of determining a velocity of the relative movement between 
at least the portion* of the pointing device and the surface. 

20 The method of Claim 19, further comprising the step of determining whether to load the new refa-ence frame Into 
' the buffer memory as a function of the velocity of the relative movement, said velocity being detenninative of 

whether a meai^ngful cross correlation can be determined wtth successive comparison frames. 

21 The method of Oaim 1 9. further comprising the step of loading the new reference frame into the buffer memory 
' before the relative movement between at least the portion of the pointing device and the surface is sufficient to 

cause a subsequent comparison frame to no longer overtap the reference frame previously stored In the buffer 
memory. 

22. The method of Claim 13. further comprising the step of filtering the Image data before detemilning the cross cor- 
relation. 

23. The method of Claim 13, wherein the image data for the reference frame are shifted. through the buffer memory a 
pixel at a time, as the cross correlation is determined using each successive pixel of the comparison frame. 

24 The method of Qaim 18. further comprising the step of providing additional temporary storage in the buffer memory 
' for a last portion of the image data of a previous reference frame, so that the last portion of the Image data of the 
previous reference frame remain available for use in determining the cross correlation as the new reference frame 
is loaded into the buffer memory. 

25. The method of Oaim 24, further comprising the step of discarding successive pbcel image data for the previous 
reference frame as the new reference frame is loaded into tte buffer memory. 

28. A machine readable medium that stores a plurality of machine Instojctions for perfonning the steps of aalm 13. 

27, A machine readable medium that stores a plurality of machine instructions for perfonning the steps of Qaim 14. 

28. A user input device that is movable and which produces an output signal indicative of a movement of the user 
input device, for input to a computing device, comprising: 

(a) a housing that Is movable relative to a surface; 

(b) an image sensor disposed in the housing and oriented so as to detect an image of the surface, producing 
a signal corresponding to pixel data for the image of the surface; 

(c) a buffer memory in which (rfxel data for a reference frame are stored, pUel data for said reference frame 
being sequentially shifted through the buffer menwry; and 

(d) a processing circuit that is coupled to the image sensor and the buffer memory, said processing circuit 
implementing a plurality of funcMons, including: 

(i) selectively loading the pixel data for the reference frame Into the buffer memory; 

(ii) determining a cross correlation between the reference frame and pixel data received from the image 
sensor as the user Input device Is moved over the surface; and 

(iii) based upon the cross correlaUon. detemiining the movement of the user input device over the surface, 
producing a conesponding output signal indicative of said movement 
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29 The user input device of Claim 28, wherein the processing circuit further determines an auto coirelatlon for the 
reference frame and modifies the output signal as a function of the auto correlation to include a sulH>ixel interpo- 
lation for the movement of the user Input device retetive to the surface. 

30. The user input device of Claim 28, further comprising a light source disposed in the housing and oriented to Illu- 
minate the surface to enable the Image sensor to deled the image of the surface. 

31 The user input device of Oairn 28, further comprising a multiplexer that is coupled to the processing circuit said 
muttiplexer Including an output coupled to an input of the iHjffer memory, a first input coupled to the Image sensor, 
and a second Input coupled to an output of the buffer memory, said multiplexer being controlled by the processing 
circuit to selectlveiy determine whether the reference frame in the buffer memory is reloaded into the Iwjffer memory, 
or that the comparison frame Is loaded Into the buffer memory as a new reference frame. 

32 The user input device of Claim 31, wherein the processing circuit causes the multiplexer to load the new reference 
* frame If the movement of the user input device relative to the surface is al>out to result in a successive comparison 

frame no longer overtapping the reference frame that Is currently stored In the buffer memory. 

33 The user input device of Claim 32. wherein the processing circuit detemnines a velocity with which the user input 
device Is moving relative to the surface to determine when to cause the multiplexer to load the new reference 
frame into the buffer memory. 

34. The user input device of aalm 28, wherein the buffer memory is unable to store aUof the pfacel data 
frame and the comparison frame, at the same time. 

35. The user input devtoe of Oalm 28. further comprising a memory in which components of the cross correlation are 
stored while the cross correlation is being determined for the comparison frame. 

36. The user input device of Claim 28, wherein cmly pixel data within a correlation window are used in determining the 
aoss correlation. 

37. A motion sensing circuit for detecting movement relative to a surface, comprising: 

(a) an image s«tsor that produces a signal corresponding to pixel data for an image of ttie surface; 

(b) a shift memory in which the pixel data are selectively loaded; 

(c) a processing circuit that controls loading of pixel data into the shift memory and determines a cross corre- 
lation between the data previously loaded into the shift memory and pbcel data for a subsequent image of the 
surface, the pixel data for successive pixels of the subsequent image being used to determine the cross cor- 
relation as the pixel data in the shift memory are sequentially moved through the shift memory, the relative 
movement being determined based upon the cross correlation thus determined. 

38 The motion sensing circuit of Claim 37, wherein the processing circuit reloads the shift memory vwth the pbcel data 
previously loaded therein unless sufficient movement has occurred to require that new pixel data be loaded mto 
the shift memory from the image sensor for use In subsequently determining the cross correlation. 

39. The motion sensing circuit of Qaim 37, w^herein the processing circuit also detennines an auto correlation for the 
pixel data loaded Into the shift memory. 

40. The motion sensing circuit of Claim 39, wherein the processing crcuit further determines a sub^xsl Interpolation 
for the relative movement as a function of the auto comelation. 

41 The motion sensing circuit of Oairn 38. further comprising a switching device that is coupled to the shift memory 
and controlled by the processing circuit to either reload the pixel data were previously loaded into the shift memory. 
oc to load the new pixel data Into Uie shift memory. 

42. The motion sensing circuit of Oairn 41 . wherein the processing circuit determines a velocity for the relative move- 
ment as a basis for detennining when to load tt^e new pixel data into the shift memory, said velocity being deter- 
minative of whether a meaningful C3t)S5 correlation can be determined vnth successive comparison frames. 
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43 The motion sensing drcuft of Claim 38. wherein the shift memory Is of suffidenl size to store one frame of pixel 
data and a portion of the frame of pixel data, said portion being used to temporarily store a portion of me pixel data 
previously loaded into the shift memory for use with the new pixel data being loaded into the shift memory for 
determining the cross correlatton. 

44. The motion sensing circuit of Claim 37, further comprising a fPter that is coupled to the Image sensor for fOtertng 
the pixel data from the image sensor. 

45. A method for determining a Bub-pbcel interpolation component for relative movement between an ot^ect and an 
adjacent surface, comprising the steps of 

(a) producing a signal corresponding to a pixelated Image of the surface: _ ^ ^ , , ^ , , , ^ 

(b) storing a reference frame for the surface that indudes data for each of a plurality of pUels of the pUelated 
image, said reterence frame being stored in a memory buffer that enables the data to be shifted through the 
memory txifffer a pixel at a time; 

(c) as relative movement occurs between the object and the adjacent surface, producing a comparison frame 
that includes data for a plurality of pixels in a new pixelated image; 

(d) detemiining a cross correlation between the data for the reference frame and the data for the companson 
frame as successive pixels of the data for the comparison frame are read from the pixilated image correspond- 
ing to the comparison frame an as said data for the reference f^me are shifted through the memoiy buffen 

(e) detennining an auto conflation of the data for the reference frame: and 

(0 determining the sub-pixel interpolation component for the relative movement as a funcUon of both the cross 
correlation and the auto correlation. 

46. The method of Claim 45. further comprising the step of detennining a pixel component of the relative motbn as a 
function of the cross correlation. 

47 The method of Claim 46. further comprising the step of selecting one of a plurality of different functions to use in 
' detennining the subi>ixel interpolation component, based upon a direction in which lh« rRUtiv« movenr^ent has 
30 occurred. 

48. A machine readable medium that stores a plurality of machine instnictions fw performing the steps of Claim 45. 

49. A machine readable medium that stores a plurality of machine Instmctions for performing the steps of Claim 47. 

50. A user controlled object that determines a sub-pixel interpolation component for relative movement between said 
object and an adjacent surface, comprising: 

(a) a pbcelated image sensor that produces a signal corresponding to pixels of an image of a surface; 

(b) a memory In which data corresponding to the pixels of the image are stored, said data compnsing a refer- 
ence frame and being shifted through the memory; . ^ .* 

(c) a processing circuit that is coupled to the memory and to the pixelated image sensor, said processing circuit 
implementing a plurality of funcUons. including: 

(i) detennining a cross correlation between the data for the reference frame stored In the memory and 
data for successive pixels of a comparison frame that corresponds to new data for the image of the surface, 
as the relative movement between the surface and the object occurs, said cross correlation being deter- 
mined as successive pixels of the comparison frame are read from the pixelated Image sensor and as 
pixels of the data for the reference frame are shifted through the memory. 

(tl) determining an auto correlation u^ the data for the reference frame that is stored in the memory; and 
(ut) detennining the sub-pbcel Interpolation component ft)r the relative movement as a function of both the 
cross conretation and the auto correlation. 

51. The system of Oalm 50. wherein the processing circuit further detemrtines a pixel component of the relative motion 
55 as a function of the cross coaelation. 

52. The system of Oaim 50. wherein the processing circuit further selects one of a plurality of different functions to usein 
detemiiningthe subi)ixel Interpolation component, based upon a direction in whichthe relative movement has occurred. 
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